ࡱ> <>;@ bjbjצצ "*82$jR444T T T 4 T 4T T V@  ^>"' 083x`Lb6T ,pDJ Index/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion Description Routine Redundant Operations: Data stored on an ISV Blade is also found on two other ISV Blades for both redundancy and processing efficiency. As an ISV Blade processes data it replicates the pertinent data on its two companion nodes over the Communications Backchannel(s) and also stores it to the SAN. A heartbeat/synchronization function between these three ISV Blades over this same Backchannel is used to provide an ISV healthy indication and synchronization among the set of three. Note that though each users data is replicated on three machines, which three machines varies for each user. As a result, machines are not necessarily companioned with any particular other machinesjust the per-user data. If a machine goes down, two machines might have to pick up the load for one user metadata index, but another two machines could pick up the load for another user metadata index. Failed or Turned ISV Blade: If an ISV Blade fails to complete a data transaction or a heartbeat sequence the other ISVs interact to replace the downed nodes functionality. There are two phases that a failure can be identified in: Data Communications: If an ISV fails or is shut down/reset in the middle of transmitting data an attempt is made to reestablish communications by one or all of the remaining nodes. The system will never lose Index or Profile Data but search/view could be lost and, at worst case, have to be retried. If communications are reestablished the interrupted node is asked to retransmit the data. If this is a recurring event at some point an Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Blade by one or more ISV Blades in the set. If communications are not reestablished the remaining two nodes will attempt to seek out another node willing to take on an additional load and then redistribute the failed nodes workload to the new node. If all available nodes are booked up the two remaining nodes will carry the load. An Exception Message is sent back to the VAPI by one or both of the remaining set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. Heartbeat Sequence: A failure of an ISV Blade to initiate or complete a Heartbeat Sequence in a timely manner will instigate a series of retry attempts based on time-stamped tasks on the SAN. If another Blade notices that an index has exceeded a preset duration it can send a ping on the Backchannel to the responsible Blade. If no response is returned then the lock and in-progress index is cancelled and the Index request resubmitted. An attempt is made to reestablish the Heartbeat. If a Heartbeat is reestablished the node is asked to rejoin the set and complete any unfinished tasks it has pending. If this is a recurring event at some point an Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Blade by one or more ISV Blades in the set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. If a Heartbeat is not reestablished the remaining two nodes seek out another node willing to take on an additional load and then redistribute the failed nodes workload to the new node. If all available nodes are booked up the two remaining nodes will carry the load. An Exception Message is sent to the SNMP engine indicating a malfunctioning/misbehaving ISV Blade by one or more ISV Blades in the set. This implies a certain level of cooperation between the remaining nodes to accomplish this task. Fail-Back/Dynamic Expansion: Since Fail-Back and Dynamic Expansion have the same functional implications we will treat them here in the same scenario. Fail-Back is defined as the action required when an ISV Blade is replaced after failure or shut down/reset. Dynamic Expansion is defined as the addition of a new ISV Blade to the array to increase performance and/or capacity. As you will see the resulting actions on the part of the ISV Blade Array members are the same for both scenarios: Status Quo Situation: The ISV Array is in Status Quo or has made accommodations for a lost element by dividing up the elements workload among two or three of the remaining ISV Blades. In either case we are adding a new resource to the array. There should never be less than two nodes responsible for user data. Resource Add Scenario: A new ISV Blade is plugged into the network switches and SAN front-end and powered up. The new ISV Blade broadcasts its presence and does a discovery of its environment. At the same time the other attached nodes, MSG Machines, and the SAN front-end see the new arrival and make a note of it in their routing tables. The new ISV Blade then searches out another ISV set that could use some offloading and negotiates a sharing of the user load: Incomplete set of two nodes available: These two other ISV Blades download their user tables and the new node takes its position as a member of the new set. Heartbeat and data communications resume as normal. Only full triad sets available: If all of the ISV Blades in the array are members of a complete set (3 ISVs) then one set will relinquish a node and the new node will pair with it for a set of 2. The set relinquishing the node will transfer ownership of a certain number of users to the exiting node and copy those to the new node also. Heartbeat and data communications resume as normal. (EFQRST\frs  ( A B yt h$5h$B*phh$h$B*phhx]hv(hXh$h0ehv(5 hNqd5h0ehX5hShhShw5 hwr5hShX5hShS5hSh5hwrhwr5CJ aJ hwr5CJ aJ hwrhX5CJ aJ -RSTsA B F & Fgdg p^pgdv( & Fgdv(^gdv(gd$^gdv($a$gdS   / e v v I$=Mp-@JS<=FGXYZ[eg ,?whyhxh/hx]h0eh5hJAth<*^hg hhg>hShE xh0ehv(5hv(hNqdhXh$GFG<=[)*de67  ^gdg> & F ^gdg>gdy & Fgdy^gd/ & FgdM^gdx]gdx] & Fgdg p^pgdg ()*4?ce{O^57]^dhnw )8A[hv(h3H hy5hg>hMh$h!h DhMh/5 h D5hhyhxh/4 1h/ =!"#$%@@@ NormalCJ_HaJmH sH tH DA@D Default Paragraph FontRi@R  Table Normal4 l4a (k@(No ListNC`N $Body Text Indent ^ B*ph*RSTsAB F G <=[)*de67  00000000000000 00 0000 00 00000 00 00 00 0x F_El\_E<9_Ee@_E?_E8l  o>*urn:schemas-microsoft-com:office:smarttags PersonName  3s@LLevv=Mi  S w*4c45nw 8AZZ[^V' CMB (P=h^`OJQJo(hHhpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hHh ^`o(hH.hpp^p`OJQJ^Jo(hHoh@ @ ^@ `OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHhPP^P`OJQJ^Jo(hHoh  ^ `OJQJo(hHh@ @ ^@ `OJQJo(hHh^`OJQJ^Jo(hHoh^`OJQJo(hHh^`OJQJo(hHh^`OJQJ^Jo(hHohPP^P`OJQJo(hHh  ^ `OJQJo(hHh^`OJQJ^Jo(hHoh!!^!`OJQJo(hH(P^V'MB                           xww{Pk' Sr i Zv I* sxl/yg>!S"d5& 'S'V^)q,^-%0/4f,6 7v9e=?@?iA0B DfD"FZHP}JH!LOGP'Q+QS8]x]<*^Nqd0eZGgiiTpp"s,NsJAtVu:CvE xy!z{d)~mSBKa<H2]W g vRwr3Hyp42$Do'"y,? ~hBJ*y MtX3e~XjF L"`~XRz-.:!gT3xYb>-n<v(D@ 9 P@UnknownGz Times New Roman5Symbol3& z Arial?5 z Courier New;Wingdings"h7Y '7Y '!24 3QH)?]WMIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion ScenariosPatti and MikePatti and Mike   Oh+'0 ,@ Xd   NIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion ScenarioslndePatti and Mikeeattatt Normal.dotiPatti and Mikee18tMicrosoft Word 10.0@@ڹ@N7Y՜.+,04 hp|  /' O NIndex/Search/View (ISV) Node Fail-Over, Fail-Back/Dynamic Expansion Scenarios Title  !"#$%&'()*,-./012456789:=Root Entry F(?Data 1TableWordDocument"*SummaryInformation(+DocumentSummaryInformation83CompObjj  FMicrosoft Word Document MSWordDocWord.Document.89q